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EXECUTIVE SUMMARY 



This report presents student-level achievement results for two groups of schools 
in Tennessee: (a) four charter schools that began operation in 2003-2004-Circles of 
Success in Learning Academy (COSLA), Memphis Academy of Health Sciences 
(MAHS), Memphis Academy of Science and Engineering (MASE), Smithson-Craighead 
Academy-and (b) three new charter schools that were established in 2004-2005-City 
University School of Liberal Arts (CityU), Star Academy, and Yo! Academy. These two 
cohorts were therefore completing their second and first academic years, respectively, 
at the time the achievement tests were administered in spring 2005. 

To conduct a rigorous and valid analysis of student achievement outcomes, we 
employed a matched program-control design at the student level in six of these seven 
schools. In this design, each charter school student was paired to a comparable 
“control” student who attended the same or a similar district school in the year prior to 
the former’s charter school enrollment. In the sixth school, which comprised primary 
elementary grades only, pretest scores were unavailable for the majority of charter 
school students and potential matched pairs. Therefore, a comparison group was 
established by randomly selecting students who were identical in race and poverty 
status to the charter students and who attended comparable neighborhood schools. 
This report supplements an earlier report (Ross, McDonald, & Bol, 2005) on the 
implementation progress made by the charter schools, encompassing school climate, 
classroom teaching methods, and perceptions by teachers, principals, parents, and 
students. 

Achievement Measures 

The Tennessee Comprehensive Assessment Program: Achievement Test 
(TCAP/AT) scores were used to assess academic achievement of students who were in 
grades 2 to 8 during the 2004-2005 school year. For students in grades 9 to 12 during 
the 2004-2005 school year, the Tennessee Gateway Assessments were used as 
outcome assessments. According to the Tennessee High School Examinations Policy, 
the State Board requires that in order to earn a high school diploma, students must 
successfully pass assessments in the following three subject areas: Algebra I (usually 
completed in grade 9), English 10, and Biology (usually completed in 10 th grade). 
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Summary and Conclusions 

Overall, the analyses of achievement found mostly positive effects for the 
second-year charter schools but equivocal outcomes for the first-year schools. 
Regarding the latter, only Star Academy but neither of the two high schools (CityU or 
Yo! Academy) demonstrated significant 1 advantages or strong effect sizes relative to 
the control groups. 

Results for the second-year Memphis schools were relatively consistent and 
clearly positive. Out of 18 school grade-level cohort x TCAP/AT subtest analyses, all 
effect sizes 2 were positive, with the median d s = +0.38 in Reading/Language Arts and 
+0.43 in Mathematics. Effect sizes of these levels would be considered educationally 
strong and meaningful by researchers. Further, of these 18 analyses, 12 were 
statistically significant at p < .05. The implication is that students attending these 
charter schools are performing better than their peers who remained in traditional 
schools. In the Nashville school (Smithson-Craighead Academy), however, no 
statistically significant effects were found in Reading/Language Arts or Mathematics 
either of the grades examined (3 rd and 4 th ), and the effect sizes were minimal. 

As indicated in the first-year report (Ross, McDonald, & McSparrin-Gallagher, 
2005), we encourage readers to interpret the results cautiously given that because of 
student choice and other constraints, we were unable to conduct a randomized 
experimental study that eliminated family interest or involvement as an influential factor, 
and (c) some grade-level matched-pair sample sizes (e.g., in COSLA, Smithson- 
Craighead, and Star Academy) were small and thus subject to sampling error. 

To capsule the achievement outcomes obtained in this study, a brief 
achievement profile of each of the six schools is provided below: 

First-Year Schools 



1 Statistical “significance” indicates effects considered “sufficiently” reliable, inferably due to actual differences and 
not due to chance (or sampling error). 

2 Effect sizes were computed by a formula called “Cohen’s d.” Each effect size (or d) indicates the number of 
standard deviations by which the charter school student mean differs from the control student mean. Thus, a effect 
size of say, +0.50, would indicate a half of a standard deviation advantage — a highly substantial educational impact. 
Generally, in education, effect sizes exceeding +0.20 would be considered meaningful and fair strong when obtained 
for a whole- school intervention. 
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City University School of Liberal Arts. Only Algebra I test scores could be 
analyzed; no effects or trends were found. 

Star Academy. Significant or suggestive positive effects were found in both 
Reading/Language Arts and Mathematics. 

Yo! Academy. No effects were found in English 10. Nor were effects found in 
Algebra I in grade 1 1 , but a nonsignificant disadvantage in Algebra I was indicated in 
grade 1 0. 

Second-Year Schools 

Circles of Success in Learning Academy. Significant positive effects were 
evidenced in grade 1 Reading/Language Arts and Mathematics, and suggestive effects 
in grade 2 Mathematics. No effects were found in grade 3. 

Memphis Academy of Health Sciences. Significant and moderate-to-strong 
positive effects were found across all grades 6 and 7 cohorts in both Reading/Language 
Arts and Mathematics. 

Memphis Academy of Science and Engineering. Significant or suggestive 
positive effects were found across all grade 7 and 8 cohorts in both Reading/Language 
Arts and Mathematics. Effects were relatively more pronounced in Reading/Language 
Arts than in Mathematics, and for the 8 th grade-Year 2 cohort than for the Year 1 7 th and 
8 th grade cohorts. 

Smithson-Craighead Academy. No effects were found in either 
Reading/Language Arts or Mathematics in either grade 3 or grade 4. However, slight 
directional advantages were evidenced for SCA students in Reading/Language Arts in 
both grades and in Mathematics in grade 4. 
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STUDENT-LEVEL ANALYSIS OF 2004-2005 ACHIEVEMENT OUTCOMES 
FOR TENNESSEE CHARTER SCHOOLS 

This report presents student-level achievement results for the four Tennessee 
charter schools that began operation in TN in 2003-2004, and three new schools that 
were established in 2004-2005. (Table 1 presents an overview of the schools by cohort, 
and a more detailed, narrative description of each school follows the table.) These two 
cohorts were therefore completing their second and first academic years, respectively, 
at the time the achievement tests were administered in spring 2005. To conduct a 
rigorous and valid analysis of student achievement outcomes, we employed a matched 
program-control design at the student level in six of these schools. In this design, each 
charter school student was paired to a comparable “control” student who attended the 
same or a similar district school in the year prior to the former’s charter school 
enrollment. Such a design has the critical advantages of controlling for pre-program 
achievement and other relevant student and school variables. In the seventh school, 
which comprised primary elementary grades only, pretest scores were unavailable for 
the majority of charter school students and potential matched pairs. Therefore, a 
comparison group was established by randomly selecting students who were identical in 
race and poverty status to the charter students and who attended comparable 
neighborhood schools. This report supplements a previous report (Ross, McDonald, & 
Bol, 2005) on the implementation progress made by the charter schools, including 
school climate, classroom teaching methods, and perceptions by teachers, principals, 
parents, and students. 
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Table 1. 

Overview of Schools 



School 


Level 


Grades 


Enrollment 


Theme 


Year 1 : 2004-2005 










City University of School of 
Liberal Arts (CityU) 


Secondary 


9 


118 


College preparation 


Star Academy 


Elementary 


K-2 


116 


Reading and language arts 


Yo! Academy 


Secondary 


10-12 


123 


Integration of visual and 
performing arts 


Year 2: 2003-2004 










Circles of Success Learning 
Academy (COSLA) 


Elementary 


K-3 


72 


Reading and language arts 


Memphis Academy of Health 
Sciences (MAHS) 


Middle 


6-7 


180 


Integration of health sciences 


Memphis Academy of Science 
& Engineering (MASE) 


Middle 


7-8 


215 


Science and engineering 
careers 


Smithson-Craighead Academy 


Elementary 


K-4 


148 


Academic and social needs of 
at-risk students 



Description of the Charter Schools 

Year 1 Schools 

City University School of Liberal Arts. City University School of Liberal Arts 
(CityU) is located in an urban area of Memphis, Tennessee, and is housed in Greater 
Middle Baptist Church. In its first year of operation, only grade 9 was established. 
African-American students constituted 98.26% of the school’s population (there was 
also one Caucasian and one Hispanic student). Approximately 70% of students were 
eligible for free or reduced-price lunches. 

The school’s faculty and staff consisted of eight teachers, one principal, one 
office manager, one instructional leader, and one part-time counselor. The 
administration reported that the teacher-to-student ratio for core classes was 1 :20 and 
for non-core classes was 1 :30. The curriculum developed and employed at CityU has a 
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liberal arts core and is aligned with College Board standards to build a strong college- 
oriented focus. This theme is continued throughout CityU’s operations, with teachers 
being referred to as “professors” and students as “scholars.” Students change classes 
after each lecture is finished, not when a bell rings, and school policies reflect a more 
collegiate atmosphere (such as allowing cell-phone usage between classes). The 
educators at CityU described their primary goal as preparing students for college and 
providing them with an opportunity to learn in a highly positive atmosphere. 

Star Academy. Star Academy is located in the northwest region of Shelby 
County on the outskirts of Memphis, Tennessee. The urban community maintains a 
variety of industries. Star Academy serves students in kindergarten through 2 nd grade. 
The student population is predominantly (98%) African-American, with approximately 
90% of the students qualifying for free or reduced-price lunch. 

In 2004-2005, there were six classroom teachers at Star Academy, plus two 
other professional staff members. The latter included a Title I Coordinator and 
Curriculum Coordinator. There is a 20:1 student-teacher ratio across classrooms. All 
professional faculty were licensed in their respective areas of instruction. In its first year 
of operation, Star Academy’s primary area of focus was Reading and Language Arts. 
The McGraw-Hill series comprised the core curricular materials in reading and math. 

For students identified as needing extra support, supplementary programs were used 
(Head Sprout and Failure Free Reading). A variety of instructional strategies, such as 
cooperative learning, peer tutoring, and direct instruction, were employed 

Yo! Academy. Yo! Academy of Visual and Performing Arts (“the Academy” or 
“Yo! Academy”) is located in southwest Memphis in an industrial warehouse area of the 
city. At the start of its first year of operation, the school held classes off-site. Academic 
classes met in space leased from a church approximately 6 miles north of the school 
building, while visual and performing arts classes met in the facility that housed the 
city’s Youth Opportunity Movement (see below) program. By midyear, the school’s own 
facility was adapted for secondary school use so that all classes could be conducted 
under a single roof. The school operated on a year-round calendar that included one 
Saturday per month. 
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During the first year of operation, the school served only students in tenth, 
eleventh, and twelfth grades. The school is scheduled to add ninth grade in its second 
year of operation. All students enrolled in 2004-2005 were African-American and 
approximately 90% were eligible for free or reduced-price meals. Nearly half of the 
students (49%) were tenth graders, over a third (37%) were eleventh graders, and the 
remainder (14%) were twelfth graders. 

The faculty consisted of one principal, one assistant principal, seven teachers, 
two office staff members, and a case manager/psychologist. All were African-American. 
Both the principal and assistant principal also taught classes. All of the teachers had 
teaching certification, and the current principal was in the final stages of completing a 
doctoral degree in school leadership. Two teachers taught one course in which they 
were not certified. Since the beginning of the school year, the school has had four 
different principals. In addition, the original executive director of the sponsoring agency 
(Yo! Memphis Foundation) was replaced. 

The mission of the school is to provide a rigorous academic preparation for 
students while incorporating the visual and performing arts, all in a safe environment for 
youths who are considered to be at-risk. Yo! Academy was established in Memphis 
under the aegis of a 5-year demonstration program funded by the U. S. Department of 
Labor called the Youth Opportunity Movement. Funding for that program concluded 
June 30, 2005. The school’s focus on the visual and performing arts is a direct 
outgrowth of the success of the Youth Opportunity Memphis (“Yo! Memphis”) program. 
In the Academy’s first year, over half of its students had experience with Yo! Memphis. 
The director of the Yo! Memphis Show Choir continued in that capacity at the Academy 
while serving as the school’s assistant principal. In 2004-2005, the program director for 
Yo! Memphis was also the executive director of the Yo! Memphis Foundation, which is 
the sponsoring agency for Yo! Academy. She also served as the school’s principal. 
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Year 2 Schools 



Circles of Success in Learning Academy (COSLA). The Circles of Success 
Learning Academy (COSLA) is an urban charter school located in Memphis, 

Tennessee. During the 2004-2005 school year, COSLA served students in grades K-3, 
91 .6% of whom were eligible for free or reduced-price lunch. 

COSLA's staff consisted of four teachers, with a teacher-student ratio of 1 :18. 
Other staff members included a principal, a curriculum facilitator, a Title I coordinator, 
an administrative assistant, and three teacher assistants. The general focus at COSLA 
is literacy development, which its school-wide program attempts to integrate across 
subject areas. The school uses scientifically-based methods and strategies in the 
delivery of the curriculum. Accordingly, curriculum content and performance standards 
are aligned with current school district and state assessment goals. The COSLA 
program also emphasizes use of a variety of teaching strategies. Organizational 
structures are designed to facilitate high levels of learning and therefore include the 
systematic monitoring of grade-level planning, classroom teaching practices, and 
student progress. Support structures include the incorporation of feedback from the 
principal and the curriculum facilitator, modeling of effective practices, provision of 
regular professional development opportunities for all staff as well as internal and 
external support services. COSLA uses the results of its annual evaluations to plan 
improvements for the upcoming year. 

Memphis Academy of Health Sciences (MAHS). The Memphis Academy of 
Health Sciences (MAHS) is located in an urban area of Memphis, Tennessee, in Shelby 
County. Specifically, MAHS is housed in the upper floor of Caldwell Elementary School. 
In 2003-2004, it served grade 6 only; in 2004-2005, grade 7 was added. African 
American students made up 100% of the school’s population. Approximately 75% of the 
students received free or reduced-price lunches. 

The school’s faculty and staff consisted of 12 teachers (including one special 
education teacher), one principal, and one secretary. The administration reported the 
teacher-student ratio as 1 :20. The curriculum is a standards-based, interdisciplinary 
program that incorporates projects and experiential learning centered on a health 
science theme. Benchmarks established by the school for its second year of operation 
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primarily focused on establishing structures that enable effective implementation and 
evaluation of the instructional program, productive communication among stakeholders, 
and a positive environment for teaching and learning. 

Memphis Academy of Science and Engineering (MASE). The Memphis 
Academy of Science and Engineering (MASE) is housed in the two upper floors of an 
office building in downtown Memphis. The seventh grade inhabits one floor and eighth 
grade the floor above. Students use stairs to travel between the floors. 

During the 2004-2005 school year, grade 8 was added to the grade 7 established 
in 2003-2004. The students were predominantly African-American (97%) with a few 
White (2.3%) and Hispanic (0.7%) students. The gender distribution was 47.1% girls 
and 52.9% boys. Free or reduced-priced lunches were provided for 68% of the 
students. The school staff in 2004-2005 included 12 teachers, two support individuals, 
three administrators, and one employee having both administrative and support duties. 
The administration consisted of a principal, a vice principal, and an educational 
specialist. 

The program includes longer days, Saturday school, high technology use, 
continuous monitoring of student progress, and an intense focus on core curricula 
integrated across subjects. Mathematics, science, and language arts are emphasized 
using authentic, varied, and integrated instruction and assessment strategies. The 
schedule entails a full day of school (8 am -5 pm) so that all classes can meet twice. In 
the afternoon, the students extended, practiced, finished, or repeated the morning’s 
lessons. Community projects (excursions) were added to allow for students to 
experience real-life applications of their classroom work. 

Smithson-Craighead Academy (SCA). Smithson-Craighead Academy, an urban 
school in Nashville, TN, serves 148 students in kindergarten through fourth grade. The 
students are predominantly (99%) African-American. All students are eligible for free 
(90%) or reduced-price (10%) lunches. The student body is classified by the 
administrators and teachers as “at-risk.” 

Staff and faculty of the school include 10 teachers and 5 teaching assistants. 
Other staff members include the principal, a curriculum coordinator and four specialists: 
librarian, special education teacher, computer instructor, and computer/special 
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education coordinator. The founder, Sister Sandra Smithson, provides administrative 
and instructional consultation. In the spring, 2005, an additional staff member was hired 
to serve as a liaison between parents and school personnel. The majority (73.7%) of 
faculty/staff reported that their education culminated in either a bachelor’s (31 .6%) or a 
master’s (42.1%) degree. Experience as a school employee ranged from 5 years or less 
(57.9%) to over 20 years (21 .1%). Over half (68.4%) have had 1-10 years of 
experience. The same percentage (68.4%) have had more than one year of experience 
at SCA. 

The focus of the school is to meet the academic and social needs of at-risk children. 
Students are taught three goals: self-control, work before play, and obey your teachers. 
They recite these goals on a daily basis. The curriculum for the lower grades includes 
benchmark testing to ensure mastery of basic skills before proceeding to more 
advanced skills. The scope and sequence of the curriculum is based on both research- 
supported practices and state curriculum standards. Teachers are given the latitude to 
use innovative strategies to meet the school’s educational goals. A daily, after-school 
program is available to provide students with additional educational support. Many 
teachers and administrators participate in the after-school program. 
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Method 



Design 

Given the different grade levels served and curricular objectives emphasized by 
the six charter schools, the results for each were analyzed separately. Except that used 
for COSLA (see below), each analysis used a matched program-control design at the 
student level, where charter school students were matched with non-charter students on 
all or most of seven criteria: 2003-2004 enrollment at the same traditional school that 
the charter school student had attended, grade level, race, gender, lunch status, and 
scores of prior achievement as assessed via the TCAP/AT for reading and mathematics 
(+/- 5 Normal Curve Equivalents). The matching process for Smithson-Craighead 
varied somewhat, in that the mean of TCAP/AT NCEs or scale scores was used as the 
matching variable for prior student achievement. 

In each analysis, a small percentage of students were not able to be matched on 
all criteria. In those instances, matches were made on three criteria: prior achievement, 
grade level, and race. Accordingly, for a small number of pairs lunch status or gender 
differed. When an appropriate match for a charter school student could not be found 
from his/her former (2002-2003 or 2003-2004) school, the closest match from one of the 
schools formerly attended by his/her present classmates was selected using all other 
criteria. Group equivalence on achievement variables at baseline was confirmed with 
one-way ANOVAs for each subject area within each school sample. Effect sizes were 
calculated on these pre-program achievement scores to confirm the similarities between 
charter and control student groups. Correlations of pre- to post-implementation 
achievement scores were also carried out by subject area within each school sample. 
Further details of the matching process for each school are included in the Results 
section (below). 

Because achievement tests differed by grade level, we conducted separate 
analyses for each grade, with one exception (Yo! Academy), using Analysis of 
Covariance (ANCOVA) or Multivariate Analysis of Covariance (MANCOVA) where prior 
achievement (pretest) scores were available, or ANOVA or MANOVA where there were 
no pretest scores. Because the same outcome measures, Algebra I Gateway scores, 
were available for the Yo! Academy tenth and eleventh graders, we conducted a two- 
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way ANCOVA in which the independent variables were grade level (10 and 11) and 
program (charter or control). Effect sizes were calculated for both unadjusted and 
adjusted mean differences within each subject area within each school sample. 

Achievement Measures 

Assessments of academic achievement utilized the TCAP/AT scores for students 
who were in grades 2 to 8 during the 2004-2005 school year. The TCAP/AT is 
administered each spring to students in grades 2 to 8 in all Memphis City Schools and 
to students in grades 3 to 8 state-wide. For students in grades 9 to 12 during the 2004- 
2005 school year, the Tennessee Gateway Assessments were used as outcome 
assessments. According to the Tennessee High School Examinations Policy, the State 
Board requires that in order to earn a high school diploma, students must successfully 
pass assessments in three subject areas: Algebra I (usually completed in 9 th grade), 
English 10, and Biology (usually completed in 10 th grade). The Gateway Assessments 
are criterion-referenced measures that indicate students’ proficiencies based on 
minimum passing scores for designation as Proficient or Advanced. Proficient is 
defined as an ability level equivalent to the minimum skill attainment needed to earn a 
passing grade in a given subject area. Advanced levels of proficiency are considered to 
be equivalent to earning a letter grade of B or higher in the course of interest. Students 
scoring below the minimum required for proficiency are considered Not Proficient. This 
lowest level is considered equivalent to earning a failing letter grade in the course. 

While each Gateway test consists of 55 items, the number of items that must be 
answered correctly in order to be scored at Proficient or Advanced differs by subject 
level and by the date the test was administered, as summarized below. 



Test 


Testing Period 


Proficient 
# Items % 


Advanced 
# Items % 


Algebra 1 


12/04 to 07/05 


30 


54.5 


42 


76.4 


English 10 


12/04 


26 


47.3 


40 


72.7 




05/05 


25 


45.4 


39 


70.9 




07/05 


24 


43.6 


38 


69.1 



Students first attempt these Gateway assessments while enrolled in the course 
for which they are assessed. Should a student fail the Gateway Assessment on this 
initial attempt, additional opportunities to complete the assessment are provided in the 
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summer following course completion. Students may also elect to retake the Gateway 
Assessment during one of three testing periods offered during each subsequent school 
year. 



Results 



City University School of Liberal Arts 

As indicated above in the school descriptions, City University School of Liberal 
Arts (CityU) is a college preparatory high school that began operation in 2004-2005 with 
ninth grade students only. There were a total of 1 18 students enrolled, of whom 98.3% 

{n = 1 16) were African-American (there were two Caucasian students). The student 
body was 57.6% female, with 66.9% qualifying for free or reduced-price lunch. 

For the analysis of student achievement, students’ Normal Curve Equivalent 
(NCE) scores in Reading and Mathematics from the 2003-2004 TCAP/AT were used as 
the baseline assessment. The NCE is a standard score distribution with a mean of 50 
and a standard deviation of approximately 21 . 

Because CityU students were in the ninth grade during the 2004-2005 school 
year, the only achievement outcome measure available was the Tennessee Gateway 
Algebra I tests. For all students in the analysis, only those attempts at the Gateway 
Algebra I test made during the 2004-2005 school year were used. The total number of 
items that students answered correctly was used as the measure of academic 
achievement. 

Matched-Pair Samples 

Of the 1 18 students enrolled at CityU, 44 had completed the Algebra I Gateway 
during the 2004-2005 school year. For the matching process, 31 of these 44 students 
were matched on all seven criteria. Of the remaining 13 students, 10 were matched on all 
but 2003-2004 location; 2 were discrepant on both location and meal status (charter 
students on reduced-price meals versus control students on full-price meals); and 1 was 
matched on all but lunch status (charter student = reduced price meals; control student = 
full-price). 
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Mathematics Outcomes 

Baseline comparability of the charter and control groups was confirmed with a 
one-way ANOVA of 2003-2004 TCAP/AT mathematics NCE scores (F = 0.001 , p = 
0.976). The low effect size confirmed this between-group comparability ( d= -0.006). 
Baseline descriptive statistics are found in Table 2. Correlations between pre- and post- 
implementation achievement scores showed a moderate to strong correlation of r = 
+0.62 (p < 0.001 ) (see Table 3). 
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Table 2. 

2003-2004 Baseline Achievement Scores in Reading and Math for Charter and Control Students by School and Grade 



School 


Group 




Reading 






Mathematics 




n 


M 


SD 


n 


M 


SD 


City U a 




Charter students 








44 


49.14 


10.87 




Control students 








44 


49.20 


10.50 




Effect Size 








d- -0.006 








One-way ANOVA 








MS= 0.102; F=0.001 ; p=0.976 




Star Academy 




Charter students 


16 


47.94 


19.16 


16 


40.06 


17.71 




Control students 


16 


47.19 


19.27 


16 


40.75 


18.44 




Effect Size 


Cfc0.04 






£/=- 0.039 








One-way ANOVA 


MS= 4.500; F= 0.012; p=0.913 


MS=3.781 ; F=0.01 2; p=0.91 5 




Yo! Academy 


1 0th Grade 


Charter students 


61 


51.23 


12.37 


16 


35.25 


10.29 




Control students 


61 


50.72 


11.84 


16 


34.44 


9.76 




Effect Size 


Cfc0.04 






d= 0.08 








One-way ANOVA 


/WS=7.877; F= 0.054; p=0.817 


MS= 5.281 ; F= 0.053; p=0.820 




1 1th Grade 


Charter students 








10 


21.90 


11.48 




Control students 








10 


21.40 


11.64 




Effect Size 








d=0.045 








One-way ANOVA 








MS= 1 .250; F= 0.009; p=0.924 




MAHS 


6 tn Grade (1 st Year) 


Charter students 


78 


40.31 


14.34 


78 


38.33 


14.42 




Control students 


78 


40.32 


14.11 


78 


38.26 


13.93 




Effect Size 


cfc-0.000 






d=+ 0.005 










MS=.006; F= 0.000; p=0.996 


AfS=.231 ; F=0.001 ; p=0.973 




7 th Grade (1 st Year) 


Charter students 


30 


40.63 


12.59 


30 


38.57 


15.29 




Control students 


30 


40.60 


12.73 


30 


38.73 


14.80 




Effect Size 


cfc+0.002 






£/=- 0.011 








One-way ANOVA 


M3=.017; F=0.000; p=0.992 


A4S=0.417; F= 0.002; p=0.966 




7 th Grade (2 nd Year) 


Charter students 


42 


42.76 


19.14 


42 


42.64 


14.97 




Control students 


42 


42.71 


18.54 


42 


42.71 


16.05 




Effect Size 


cfc+0.002 






d=-0.004 








One-way ANOVA 


MS=.048; F=0.000; p=0.991 


/WS=.107; F=0.000; p=0.983 
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Table 2, continued. 

2003-2004 Baseline Achievement Scores in Reading and Math for Charter and Control Students by School and Grade 



School 


Group 




Reading 






Mathematics 




n 


M 


SD 


n 


M 


SD 


MASE 


7 tn Grade (1 st Year) 


Charter students 


91 


49.77 


15.15 


91 


47.44 


16.84 




Control students 


91 


49.68 


15.21 


91 


47.35 


16.51 




Effect Size 


d=0.006 






cf=0.005 








One-way ANOVA 


MS=0.352; F =0.002; p=0.969 


MS=0. 4 1 7; F=0. 002; p=0. 966 




8 th Grade (1 st Year) 


Charter students 


25 


51.92 


14.68 


25 


50.48 


13.66 




Control students 


25 


52.20 


14.77 


25 


50.72 


13.22 




Effect Size 


cf=-0.01 9 






£/=- 0.018 








Oneway ANOVA 


MS=0.980; F =0.005; p=0.947 


MS=0. 720; F =0.004; p=0.951 




8 th Grade (2 nd Year) 


Charter students 


99 


60.31 


16.06 


99 


51.14 


15.14 




Control students 


99 


60.19 


15.87 


99 


51.75 


14.79 




Effect Size 


cf=0.008 






cf=-0.041 








One-way ANOVA 


MS=0. 727; F =0.003; p=0.957 


MS= 18. 182; F=0.081; p=0. 776 




Smithson-Craighead 


3 rd Grade (2 nd Year) 0 


Charter students 


27 


35.19 


11.30 


27 


31.78 


14.36 


NCEs 


Control students 


27 


40.44 


12.43 


27 


29.93 


17.64 




Effect Size 


d=- 0.451 






d= 0.117 








One-way ANOVA 


MS=373.407; F=2.647; p=0. 1 10 


MS=46.296; F =0.179; p=0.674 




4 th Grade (2 nd Year) 


Charter students 


26 


439.23 


39.295 


26 


438.15 


18.454 


Scale Scores 


Control students 


26 


448.08 


21.376 


26 


438.69 


23.375 




Effect Size 


£/=- 0.285 






£/=- 0.026 








One-way ANOVA 


MS=1 01 7.308; F=1 .017; p=0.318 


MS=3. 769; F=0.008; p=0.927 





a For CityU, only mathematics achievement was analyzed because English or reading posttest scores were unavailable for its ninth graders. 
b Language NCEs at baseline were not significantly different (d=0.017; MS=0.907; F=0.004; p=0.950) 
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Table 3. 

Baseline-Posttest Correlations by Charter School and Subject 



School 


Subject 


Baseline 


Posttest 


Corr. 


Prob. 


CityU 




Mathematics 


2003-2004 TCAP/AT Math NCE 


Gateway Algebra 1 number of items correct 


0.620 


<0.001 


Star Academy 




Reading 


2003-2004 TCAP/AT Reading NCE 


2004-2005 TCAP/AT Reading NCE 


0.582 


<0.001 




Mathematics 


2003-2004 TCAP/AT Math NCE 


2004-2005 TCAP/AT Math NCE 


0.622 


<0.001 


Yo! Academy 




Reading 


8th Grade TCAP/AT Reading NCE 


Gateway English 1 0 number of items correct 


0.460 


<0.001 


10 th Grade 


Mathematics 


8th Grade TCAP/AT Math NCE 


Gateway Algebra 1 number of items correct 


0.258 


0.155 


11 th Grade 


Mathematics 


8th Grade TCAP/AT Math NCE 


Gateway Algebra 1 number of items correct 


0.106 


0.656 


COSLA a 


2 nd Year 


2nd Grade 


Reading 2003-04 TCAP NCE 


Reading 2004-05 TCAP NCE 


.174 


.309 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP NCE 


.473 


.004 


2 nd Year 


3rd Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.617 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP # Correct 


.604 


<.001 


MAHS 


1 st Year 


6th Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.616 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP # Correct 


.727 


<.001 


1 st Year 


7th Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.575 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP # Correct 


.768 


<.001 


2 nd Year 


7th Grade 


Reading 2002-03 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.703 


<.001 






Math 2002-03 TCAP NCE 


Math 2004-05 TCAP # Correct 


.778 


<.001 


MASE 


1 st Year 


7th Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.744 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP # Correct 


.802 


<.001 


1 st Year 


8th Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.711 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP # Correct 


.810 


<.001 


2 nd Year 


8th Grade 


Reading 2002-03 TCAP NCE 


Reading/LA 2004-05 TCAP # Correct 


.694 


<.001 






Math 2002-03 TCAP NCE 


Math 2004-05 TCAP # Correct 


.760 


<.001 
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Table 3, continued. 

Baseline-Posttest Correlations by Charter School and Subject 



School 


Subject 


Baseline 


Posttest 


Corr. 


Prob. 


Smithson-Craighead 


2 na Year 


3 ra Grade 


Reading 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP Scale Score 


0.506 


<.001 






Language 2003-04 TCAP NCE 


Reading/LA 2004-05 TCAP Scale Score 


0.370 


<.001 






Math 2003-04 TCAP NCE 


Math 2004-05 TCAP Scale Score 


0.670 


<.001 


2 nd Year 


4 th Grade 


Reading/LA 2003-04 TCAP Scale Score 


Reading/LA 2004-05 TCAP Scale Score 


0.485 


<.001 






Math 2003-04 TCAP Scale Score 


Math 2004-05 TCAP Scale Score 


0.613 


<.001 



a COSLA pretests were not included in the achievement analyses because of small sample sizes for control students. 
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Descriptive statistics on the Gateway Algebra I “posttest” assessment for the 
CityU and control students are provided in Table 4. Standard and adjusted effect sizes 
are also given. The low adjusted effect size ( d= -0.06) is indicative of the outcomes of 
the ANCOVA (Table 4), where no significant difference was found between charter 
school students and control group students in their levels of mathematics achievement, 
F(1, 85) = 0.088, p = 0.768. As expected, the pretest covariate was highly significant, 
F(1, 85) = 53.23, p< 0.001. 
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Table 4. 

2003-2005 “Posttest” Achievement Scores in Reading and Math for Charter and Control Students by School and Grade 



School 


Group 




Reading 






Mathematics 




N 


M 


SD 


Adj. M 


N 


M 


SD 


Adj. M 


CityU a 


9 th Grade 


Charter students 










44 


38.11 


6.71 


38.13 




Control students 










44 


38.52 


8.40 


38.51 




Effect Size 










d=- 0.05 










Adj. Effect Size 










d=- 0.06 








Star Academy 0 


2 na Grade 


Charter students 


16 


46.06 


12.56 


45.91 


16 


51.50 


15.23 


51.73 




Control students 


16 


38.19 


13.33 


38.34 


16 


34.56 


19.64 


34.33 




Effect Size 


d= 0.63 








d= 0.99 










Adj. Effect Size 


d=0.74 








cfcl .42*** 








Yo! Academy c 


10 ,n Grade 


Charter students 


61 


37.89 


5.89 


37.82 


16 


29.50 


6.022 


28.81 




Control students 


61 


37.25 


6.90 


37.31 


16 


33.00 


6.643 


32.41 




Effect Size 


cfcO.IO 








d= 0.57 










Adj. Effect Size 


d= 0.09 








cfcO.548 








11 th Grade 


Charter students 










10 


26.00 


6.633 


26.99 




Control students 










10 


26.30 


6.584 


27.35 




Effect Size 










d=0.048 










Adj. Effect Size 










d=0.056 








COSLA Dd 


3 ra Grade 


Charter students 


16 


28.13 


6.67 


NA 


16 


31.25 


6.62 


NA 




Control students 


19 


27.00 


8.75 


NA 


19 


30.95 


7.67 


NA 




Effect Size 


d=0.129 








d=0.039 










Adj. Effect Size 


NA 








NA 








2 nd Grade 


Charter students 


16 


45.63 


18.73 


NA 


16 


52.94 


15.16 


NA 




Control students 


20 


45.25 


14.29 


NA 


20 


41.75 


18.85 


NA 




Effect Size 


d= 0.027 








Cfc0.594 










Adj. Effect Size 


NA 








NA 








1 st Grade 


Charter students 


19 


69.53 


20.90 


NA 


19 


54.05 


18.06 


NA 




Control students 


20 


50.50 


15.69 


NA 


20 


38.70 


18.26 


NA 




Effect Size 


cf=1 .21 3** 








d=0.841* 










Adj. Effect Size 


NA 








NA 
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Table 4, continued. 

2003-2005 “Posttest” Achievement Scores in Reading and Math for Charter and Control Students by School and Grade 



Reading Mathematics 



School 


Group 


N 


M 


SD 


Adj. M 


n 


M 


SD 


Adj. M 


mahs d 


6 m Grade (1 st Year) 


Charter students 


78 


32.64 


7.76 


32.64 


78 


34.71 


8.37 


34.69 




Control students 


78 


26.27 


8.01 


26.27 


78 


29.85 


8.30 


29.86 




Effect Size 


rf=0.792 








d= 0.586 










Adj. Effect Size 


d=0.795*** 








d=0.582*** 








7 th Grade (1 st Year) 


Charter students 


30 


31.17 


9.06 


31.18 


30 


33.23 


8.89 


33.27 




Control students 


30 


26.30 


9.66 


26.28 


30 


28.50 


7.96 


28.46 




Effect Size 


d= 0.504 








d= 0.594 










Adj. Effect Size 


rf=0.501* 








d= 0.604** 








7 th Grade (2 nd Year) 


Charter students 


42 


34.38 


8.93 


34.38 


42 


32.76 


8.51 


32.77 




Control students 


42 


27.64 


8.67 


27.64 


42 


29.50 


8.71 


29.49 




Effect Size 


cfc+0.777 








£/=+ 0.374 










Adj. Effect Size 


cfc+0.777** 








d=+ 0.376** 








MASE° 


7 m Grade (1 st Year) 


Charter students 


91 


35.68 


7.39 


35.66 


91 


38.24 


7.08 


38.31 




Control students 


91 


31.92 


9.88 


31.94 


91 


35.92 


9.19 


35.85 




Effect Size 


d= 0.381 








d=0.148 










Adj. Effect Size 


d= 0.377*** 








d=0.144 








8 th Grade (1 st Year) 


Charter students 


25 


36.16 


6.96 


36.22 


25 


38.24 


7.08 


38.31 




Control students 


25 


32.32 


7.64 


32.26 


25 


35.92 


9.19 


35.85 




Effect Size 


rf=0.503 








d= 0.252 










Adj. Effect Size 


rf=0.518* 








d= 0.268 








8 th Grade (2 nd Year) 


Charter students 


99 


35.68 


7.26 


35.71 


99 


38.13 


8.51 


38.25 




Control students 


99 


33.64 


8.78 


33.60 


99 


34.40 


9.21 


34.28 




Effect Size 


d= 0.232 








d= 0.405 










Adj. Effect Size 


d= 0.240* 








d= 0.431*** 
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Table 4, continued. 

2003-2005 “Posttest” Achievement Scores in Reading and Math for Charter and Control Students by School and Grade 



Reading Mathematics 



School Group 


N 


M 


SD 


Adj. M 


n 


M 


SD 


Adj. M 


Smithson-Craighead 


3 ra Grade (2 na Year) Charter students 


27 


469.85 


28.60 


471.96 


27 


448.74 


19.44 


448.83 


Control students 


27 


469.04 


21.58 


466.93 


27 


450.93 


24.36 


450.84 


Effect Size 


cf=0.033 








cfc-0.101 








Adj. Effect Size 


d= 0.230 








d=- 0.128 








4 th Grade (2 nd Year) Charter students 


26 


460.42 


30.95 


462.08 


26 


472.42 


27.00 


472.80 


Control students 


26 


458.81 


34.41 


457.15 


26 


463.54 


25.81 


463.16 


Effect Size 


d= 0.050 








d= 0.343 








Adj. Effect Size 


Cfc0.181 








Cfc0.465 









a Gateway Algebra I assessments were used as the follow-up measures of achievement. No reading scores were available for the CityU cohort. 
b TCAP/AT NCE scores in Reading/Language Arts and Mathematics from 2004-2005 were used as the follow-up measures. 
c Because of small sample sizes and other scores, 1 1 th grade Yo! Academy students were excluded from the English 10 analyses. 
d Because pretest scores were not available, no adjusted means or adjusted effect sizes were computed. 

* p<. 05 

** pc. 01 
*** pc. 001 
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Gateway Proficiency Levels 

Table 5 displays Gateway proficiency levels attained for charter students as 
compared to the matched control school students. Chi-square tests of between group 
differences were nonsignificant (x 2 (2) = 4.130, p = 0.127). For charter students, 93.2% 
scored either Proficient or Advanced on the Gateway Algebra I assessment, compared 
to 84.1% of control group students. 
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Table 5. 

Two-way chi-square results examining the relationship between Group (charter vs. control) and Posttest Proficiency Level 









Below Proficient 


Proficient 


Advanced 


Chi 


School 


Subject Area 


Group 


n 


% 


n 


% 


n 


% 


Square 


CityU 




Algebra 1 


Charter students 


3 


6.8 


28 


63.6 


13 


29.5 


A IQ 






Control students 


7 


15.9 


19 


43.2 


18 


40.9 


L r. 1 O 


Yo! Academy 


1 0 th 


English 10 


Charter students 


1 


1.6 


27 


44.3 


33 


54.1 


n 74Q 


1 U OldUt? 




Control students 


2 


3.3 


30 


49.2 


29 


47.5 


u. / H-y 


1 0 th r^rarlo 


Algebra 1 


Charter students 


8 


50.0 


7 


43.8 


1 


6.2 


1 999 


1 U OldUc? 




Control students 


5 


31.3 


10 


62.5 


1 


6.2 


1 .C.C.C. 


1 1 th 


Algebra 1 


Charter students 


8 


50.0 


2 


20.0 


0 


0 


n 


1 1 OlciUc 




Control students 


8 


50.0 


2 


20.0 


0 


0 


u 


MAHS 6 tn Grade 1 st 


Year 




















Reading/LA 


Charter students 


5 


6.4 


56 


71.8 


17 


21.8 


90 849** 






Control students 


20 


25.6 


56 


71.8 


2 


2.6 






Math 


Charter students 


8 


10.3 


54 


69.2 


16 


20.5 


1 9 978* 






Control students 


17 


21.8 


58 


74.4 


3 


3.8 


1 4.4 / O 


MAHS 7 tn Grade 1st Year 




Reading/LA 


Charter students 


3 


10.0 


20 


66.7 


7 


23.3 


4 Ofil 






Control students 


9 


30.0 


17 


56.7 


4 


13.3 


‘-T.WVJ 1 




Math 


Charter students 


4 


13.3 


21 


70.0 


5 


16.7 


4 814 






Control students 


9 


30.0 


20 


66.7 


1 


3.3 


H-.O 1 H- 


MAHS 7 tn Grade 2 n0 


'Year 




















Reading/LA 


Charter students 


2 


4.8 


23 


54.8 


17 


40.5 


1 8 871 ** 






Control students 


10 


23.8 


28 


66.7 


4 


9.5 


10 . 0 / 1 




Math 


Charter students 


6 


14.3 


30 


71.4 


6 


14.3 


9 887 






Control students 


12 


28.6 


24 


57.1 


6 


14.3 


4.00 / 
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Table 5, continued. 

Two-way chi-square results examining the relationship between Group (charter vs. control) and Posttest Proficiency Level 







Below Proficient 


Proficient 


Advanced 


Chi 


School Subject Area 


Group 


n 


% 


n 


% 


n 


% 


Square 


MASE 7 in Grade 1st year 


Reading/LA 


Charter students 


3 


3.3 


56 


61.5 


32 


35.2 


Q QA9* 




Control students 


14 


15.4 


53 


58.2 


24 


26.4 


O.OH-O 


Math 


Charter students 


7 


7.7 


58 


63.7 


26 


28.6 






Control students 


15 


16.5 


49 


53.8 


27 


29.7 


o . ooo 


MASE 8 in Grade 1st year 


Reading/LA 


Charter students 


0 


0.0 


16 


64.0 


9 


36.0 


P fiPf) 




Control students 


2 


8.0 


17 


68.0 


6 


24.0 




Math 


Charter students 


0 


0.0 


17 


68.0 


8 


32.0 


9 DQ7 




Control students 


2 


8.0 


16 


64.0 


7 


28.0 


d.W / 


MASE 8 m Grade 2nd year 


Reading/LA 


Charter students 


3 


3.0 


52 


52.5 


44 


44.4 






Control students 


12 


12.1 


47 


47.5 


40 


40.4 




Math 


Charter students 


4 


4.0 


52 


52.5 


43 


43.4 


o Q79* 




Control students 


9 


9.1 


66 


66.7 


24 


24.2 


o.y / d. 


Smithson-Craighead 3 ra Grade 2 na 


Year 
















Reading/LA 


Charter students 


4 


14.8 


19 


70.4 


4 


14.8 






Control students 


5 


18.5 


20 


74.1 


2 


7.4 


u.ouo 


Math 


Charter students 


11 


40.7 


14 


51.9 


2 


7.4 


i ono 




Control students 


15 


55.6 


9 


33.3 


3 


11.1 


1 .Wd 


Smithson-Craighead 4 m Grade 2 na 


Year 
















Reading/LA 


Charter students 


10 


38.5 


16 


61.5 


0 


0 


P fiOO 




Control students 


11 


44.0 


12 


48.0 


2 


8.0 


t.OUU 


Math 


Charter students 


9 


34.6 


16 


61.5 


1 


3.8 






Control students 


12 


48.0 


11 


44.0 


2 


8.0 


i .Doy 



p< .05 

p< .01 
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Star Academy 

Matched-Pair Samples 

Star Academy is a college preparatory elementary school that began its 
inaugural year with students in grades K-2. For the 2004-2005 school year, 116 
students were enrolled, comprising 39 Kindergarteners, 39 first graders, and 38 second 
graders. Overall, half of the students were male, and 99.14% were African-American. 
There was one Hispanic child in the first grade group. In addition, 87.1% of the 
students overall received free or reduced-price meals. 

Of the 1 1 6 students enrolled at Star Academy, only 1 6 of the second-grade 
students had TCAP/AT scores from the 2003-2004 school year. Of these, only 3 
students were matched with control students on all seven matching criteria. Of the 
remaining students, 10 required matches with control students from school locations 
other than their home school in 2003-2004. In addition, one male student was matched 
with one female student from a different school; one charter student on reduced-price 
meals was matched with one control on full-price meals; and one charter student 
differed from his/her control counterpart on 2003-2004 location and meal status (charter 
was reduced-price, control was on free meals). In all 16 matches, prior achievement, 
grade level, and race were matched. Because of the small number of students 
available for the analysis, separate analyses of reading and mathematics achievement 
outcomes were done via ANCOVAs as opposed to one MANCOVA. 

Reading and Mathematics Outcomes 

Baseline assessments of student achievement used students’ NCE scores in 
Reading/Language Arts and Mathematics from the 2003-2004 TCAP/AT. One-way 
ANOVAs performed on baseline achievement scores confirmed the adequacy of the 
matching process in both reading and mathematics (Reading: F= 0.012, p = 0.913; 
Math: F= 0.012, p = 0.915). Small effect sizes added further evidence of a close 
match, with d= +0.04 for Reading/Language Arts, and d= -0.039 for Mathematics. 
Baseline descriptive results are found in Table 2. Correlations between achievement 
scores baseline to post-charter school intervention were moderate to strong, with r = 
+0.582 (p < 0.001 ) for Reading/Language Arts and r = +0.622 (p < 0.001 ) for 
Mathematics (see Table 3). 
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Post-charter school achievement was based on TCAP/AT NCE scores from 
2004-2005. A depiction of these data, including both adjusted and unadjusted effect 
sizes for the mean group differences, is found in Table 4. Differences in 
Reading/Language Arts achievement were marginally significant, F( 1 ,29) = 4.154, p = 
0.051 , with a strong adjusted effect size of d = +0.744. The adjusted mean NCE for the 
Star Academy students was 45.91 versus 38.34 for the control group students. It is 
likely that the small sample size limited the power of this analysis to detect a statistically 
significant between-group difference. 

In the area of mathematics achievement, charter students demonstrated a 
significant advantage over matched controls, F{ 1 ,29) = 15.059, p < 0.001 . A 
correspondingly large adjusted effect size of d = +1 .42 suggests an educationally strong 
and meaningful effect on 2004-2005 achievement. Star Academy students’ adjusted 
mean NCE in mathematics for 2004-2005 was 51 .73 compared to control students’ 
adjusted mean NCE of only 34.33. 

Yo! Academy 
Matched-Pair Samples 

Yo! Academy, a high school for at-risk students, offers a focus on visual and 
performing arts. For the school’s first year (2004-2005), 123 tenth through twelfth grade 
students were enrolled, including 71 tenth graders, 42 eleventh graders, and 10 twelfth- 
graders. All students were African-American, and 87.8% qualified for free or reduced- 
price meals. 

As all these students were in upper grades, TCAP/AT NCEs for reading and 
mathematics for each student’s eighth-grade year were used as indicators of prior 
achievement. Therefore, the matching process entailed attempting to match on the 
following seven criteria: eighth grade TCAP/AT for reading and mathematics (+/- 5 
NCEs), 2003-2004 enrollment at the same school that the charter school student 
formerly attended, grade level, race, gender, and lunch status. 

Gateway assessments were used as posttest measures of program impact on 
student academic achievement. Only those students (both charter and control 
students) who completed either an Algebra I and/or English 10 Gateway test during the 
2004-2005 school year were considered for the analysis. The total number of items 
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answered correctly on the Gateway tests was used in each analysis. Only 20 Yo! 
Academy students with matched controls had taken both the Algebra I and English 10 
Gateway assessments. Therefore, separate ANCOVAs were performed by subject 
area, rather than employing a MANCOVA for both subject areas. 

For the analysis of reading skills, 64 Yo! Academy students had matched 
controls. Out of these 64 students, 47 were matched with controls on all targeted 
variables. Of the remaining 17 students, 8 were matched with controls in different 
locations in 2003-2004; 7 were matched controls with different meal status 
designations (3 charter students = reduced-price meals but their matched controls = 
free meals; 3 charter students = free meals but controls = full price meals; and 1 charter 
student = free meals but control = reduced-price meals); and the 2 remaining students 
were matched with controls on all variables except prior school location and meal status 
(1 charter student = reduced-price meals but control = full price; and 1 charter student = 
free meals but the control = full price). 

These 64 matched students comprised 61 tenth graders and 3 eleventh graders. 
Grade-level differences in baseline TCAP/AT NCEs were found, with tenth-grade 
charter students averaging 51 .23 ( SD = 12.37) and tenth-grade controls averaging 
50.72 (SD = 1 1 .84). In contrast, eleventh graders at Yo! Academy averaged 17.33 (SD 
= 1 1 .59) NCE on their eighth-grade TCAP/AT, whereas controls averaged 17.33 (SD = 
13.32). Given these strong grade-level differences and the extremely low eleventh- 
grade performances, the three eleventh-grade matched pairs were dropped from the 
analysis. 

For the analysis of math achievement, the study population comprised 26 Yo! 
Academy students with Algebra I Gateway scores from 2004-2005, with 1 6 students in 
tenth grade and 10 in eleventh grade. Of these, 15 students were matched with control 
students on all variables. An additional 8 charter students were matched on all but 
school location in 2003-2004, and 3 students were matched on all but meal status (1 
charter student = reduced-price meals but control = free meals; 1 charter student = free 
meals but control = full price; and 1 charter student = full price but control = free meals). 
Due to the relatively even split of the sample by grade level, descriptive data are 
summarized by grade level, and a two-way, Group (charter or control) and Grade Level 
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(10 th or 1 1 th ) ANCOVA, with eighth grade TCAP/AT Mathematics NCE as the covariate, 
was performed on the Algebra I Gateway Scores. 

Reading and Mathematics Outcomes 

Comparability of tenth-grade charter and control students’ baseline assessments 
of academic achievement in reading was verified with a one-way ANOVA performed on 
eighth-grade TCAP/AT Reading/Language Arts NCE scores (F = 0.054, p = 0.817). A 
correspondingly low effect size of d = +0.04 further confirmed the baseline group 
similarity (see Table 2). Even though 2 years had elapsed from the date of the eighth- 
grade baseline TCAP/AT achievement assessment to the time of the follow-up Gateway 
test, the correlation of TCAP/AT Reading/Language Arts NCE scores with Gateway 
English 10 scores was moderately strong (r= +0.460, p < 0.001) (see Table 3). 
Unadjusted and adjusted posttest means are shown for the charter and control groups 
in Table 4. The low adjusted effect size ( d= +0.09) is consistent with ANCOVA 
outcomes showing no significant group differences in Gateway English 10 achievement 
between charter and control students, F(1 ,119) = 0.248, p = 0.620. The pretest 
covariate was the only significant source of variance (p < 0.001). 

A summary of students’ Gateway proficiency levels for the English 10 
assessments completed during the 2004-2005 school year is provided in Table 5. A 
chi-square test of the relationship between Group (charter vs. control) and Performance 
Level was nonsignificant, x 2 (2) = 0.749, p = 0.688. Over 98% of charter students and 
96.7% of the control students score at Proficient or Advanced levels. 

For the Algebra I analysis, baseline TCAP/AT NCE scores for the charter school 
and control groups are shown in Table 2. One-way ANOVAs were performed on each 
grade level to ensure similarity on this baseline assessment (10 th grade: F= 0.053, p = 
0.820, d = +0.08; 11 th grade: F= 0.009, p = 0.924, d= +0.045). The correlation of 
baseline to follow-up assessments of mathematics achievement was weak for both the 
tenth-grade group (r= +0.258, p = 0.155) and the eleventh-grade group (r= +0.106, p = 
0.656) (see Table 3). As with the English 10 assessments, there was a 2- to 3-year lag 
between pre- and post-assessments. 

Table 4 summarizes group means and effect sizes for posttest (2004-2005) 
Gateway Algebra I assessments. The adjusted effect size for the tenth-grade sample 
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was moderately to strongly negative (adjusted d = -0.548), thus suggesting a directional 
advantage for the control group. For the eleventh graders, Gateway scores differed 
little by group (adjusted d= -0.056). In the ANCOVA, the eighth grade pretest covariate 
was not significant (p = .153), mirroring the weak pretest-posttest correlation described 
above (see Table 3). More critically, no significant main effects were found in Algebra I 
achievement either by Group, F( 1 ,47) = 1 .1 91 , p = 0.281 ), or by Grade Level, F( 1 ,47) = 
2.580, p = 0.1 15). The interaction of Group and Grade Level was also nonsignificant, 

F(1 ,47) = 0.796, p = 0.377. 

Gateway Proficiency Levels 

A summary of 2004-2005 Gateway proficiency levels for Algebra I assessments 
is provided in Table 5. For tenth graders, there was no relationship between Group 
(charter vs. control) and Performance Level, x 2 (2) = 1 .222, p = 0.543. Half (50%) of the 
tenth-grade Yo! Academy students scored at the Proficient or Advanced levels 
compared with 68.7% of the tenth-grade control students. For eleventh graders, there 
were no group differences in the number of students scoring at Proficient (8 out of 10 in 
each group) or Advanced (2 out of 10 in each). 

Circles of Success in Learning Academy (COSLA) 

COSLA and Comparison Group Samples 

Enrollment at COSLA consisted of 20 first graders, 20 second graders, and 19 
third graders. Of these, newly enrolled (first-year) students consisted of 1 first grader, 4 
second graders, and 3 third graders. Because these transfer cohorts were so small, in 
contrast to those for MAHS and MASE above, we decided not to include them in the 
analyses. In the final grade levels samples of 19, 16, and 16, all students were African 
American and qualified for free or reduced-price lunch. Given that TCAP/AT testing 
begins in grade 2 in Memphis City Schools and is optional for schools to administer in 
grade 1 , only a very small number of students had “pretest” scores reflecting their 
achievement prior to enrolling in COSLA. Therefore, it was not possible to establish a 
matched-pair “control” group as done for the other charter schools. Instead, we created 
a “comparison” group by randomly selecting African American students who qualified for 
free or reduced-price lunch from the same traditional schools that the third-grade 
COSLA students had attended (in 2002-2003) prior to their charter school enrollment. 
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Reading and Mathematics Outcomes 

Due to the small sample sizes, we conducted separate ANOVAs within grades 
on the Reading/Language Arts and Mathematics posttest scores. Posttest means, 
standard deviations, and effect sizes are summarized in Table 4. 

Posttest outcomes for grade 1. The univariate ANOVAs revealed significant 
results favoring COSLA students in Reading/Language Arts, F{ 1 ,37) = 10.41 , p = .003, 
and Mathematics, F(1 ,37) = 6.96, p = .012. Effect sizes reflected strong advantages in 
both subjects (respective ES's = +1.21 and +0.84). 

Posttest outcomes for grade 2. The ANOVAs yielded nonsignificant outcomes 
reflecting fairly comparable means ( ES = +0.03) for both groups in Reading/Language 
Arts, F{ 1 ,34) = 0.01 , p = .946, but a directional advantage (ES = +0.59) for COSLA in 
Mathematics, F(1 ,34) = 3.71 , p = .06. 

Posttest outcomes for Grade 3. The univariate ANOVAs revealed nonsignificant 
results in both Reading/Language Arts, F( 1,33) = .177, p= .68, and Mathematics, 

F(1 ,33) = 0.02, p = .90). The associated effect sizes of +0.13 and +0.04 were indicative 
of small program impacts. 

Summary of COSLA outcomes. Advantages for COSLA were statistically 
significant and strong for both Reading/Language Arts and Mathematics in grade 1 . 

The associated effect sizes were +1 .21 and +0.84, respectively. Given the small 
sample sizes, the nonsignificant grade 2 advantage for COSLA in Mathematics (+0.59) 
also seems suggestive of positive impacts. However, effects were small and close to 
zero in Reading/Language Arts in grade 2 and in both subjects in grade 3. 

Memphis Academy of Health Sciences (MAHS) 

Matched-Pair Samples 

MAHS implements a standards-based program that incorporates interdisciplinary 
projects and experiential learning centered on a health science theme. In the present, 
second year of operation it included grades 6 and 7, in which student enrollments were 
78 and 72, respectively. In the Year 1 matched-pairs achievement study, 70 matched 
sixth-grade charter-control student pairs were established. Of these, for the 2004-2005 
school year, 6 dropped out of the school district, 10 transferred to another location, 5 
had missing achievement data for 2004-2005, and 7 lost their control school matches as 
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a result of the latter failing a grade. The remaining longitudinal sample for the Year 2 
(2004-2005) seventh-grade analysis thus consisted of 42 matched pairs from Year 1 . In 
addition, 30 new seventh graders enrolled in MAHS in 2004-2005 and were matched to 
control school counterparts. Fifteen pair differed only on pretest school location. For 
the sixth-grade analysis, all 78 new enrollees were matched to control school 
counterparts. Forty pair differed on pretest school location. 

Reading and Mathematics Outcomes 

To ensure comparability of MAHS and control group cohorts, pretest NCE scores 
in Reading/Language Arts and Mathematics were compared between groups for the 6 th 
grade-Year 1 cohort, the 7 th grade-Year 1 cohort, and the 7 th grade-Year 2 cohort. As 
summarized in Table 2, the MAHS and control students, as expected, performed nearly 
identically on all tests, with all ES's approximating zero and all ANOVA outcomes 
nonsignificant. Both groups scored below the national norm of 50, with the mean NCE’s 
ranging from the high 30’s to low 40’s. As may be seen in Table 3, the pretest-posttest 
correlations for the MAHS grade-level by subject were moderately strong to strong, 
ranging from r= +.575 to +0.778. 

For the analysis of program effects on posttest scores, a multivariate analysis of 
covariance (MANCOVA) was performed, separately by cohort, on the 2004-2005 CRT 
TCAP/AT Reading/Language Arts and Mathematics subtests. Posttest means, 
standard deviations, and effect sizes are summarized in Table 4. 

Posttest outcomes for E> h grade-Year 1 cohort. The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s <.001 ). 
The multivariate effect of program was also significant, F(2,151) = 31.32, p< .001). 
Follow-up univariate ANCOVAs revealed significant results for both Reading/Language 
Arts, F(1,152) = 52.31, p<.001, and Mathematics, F(1,152) = 30.64, p< .001. Adjusted 
effect sizes for Reading/Language Arts and Mathematics were +0.795 and +0.582, 
respectively, thereby reflecting fairly strong program effects. 

Posttest outcomes for 7 th grade-Year 1 . The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s < .01). 
The multivariate effect of program was also significant, F( 2,55) = 7.20, p < .001 . 
Univariate ANCOVAs revealed significant results for both Reading/Language Arts, 
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F(1,56) = 7.16, p = .01 , and Mathematics, F( 1,56) = 13.20, p = .001. Adjusted effect 
sizes for Reading/Language Arts and Mathematics were +0.501 and +0.604, 
respectively, reflecting fairly strong program effects (see Table 4). 

Posttest outcomes for 7 th grade-Year 2. The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s < .001). 
The multivariate effect of group (MAHS or Control) was also significant, F( 2,79) = 16.98, 
p < .001 . Univariate ANCOVAs were significant for both Reading/Language Arts, 

F(1 ,80) = 33.97, p < .001 , and Mathematics, F(1 ,80) = 9.1 6, p = .003, favoring MAHS 
students. Adjusted effect sizes for Reading/Language Arts and Mathematics were 
+0.78 and +0.38, respectively, again reflecting strong program effects, especially in 
Reading/Language Arts (see Table 4). 

TCAP/AT proficiency levels. A supplementary analysis was performed to 
examine the percentages of MAHS and control students who scored at Below 
Proficient, Proficient, and Advanced levels on the spring 2004-2005 CRT 
Reading/Language Arts and Mathematics subtests. Summaries appear in Table 5. 
Two-way chi square (program x proficiency level) analyses, all favoring MAHS, were 
significant for the 7 th grade-Year 2 students in Reading/Language Arts only, and 6 th 
grade-Year 1 students in both subjects. Higher percentages of MAHS than control 
students tended to score Proficient and Advanced in both Reading/Language Arts and 
Mathematics. 

Summary of MAHS outcomes. For all three cohorts, strong program effects were 
obtained. In Reading/Language Arts, the three effects sizes were +0.795, +0.501 , and 
+0.78 for the 6 th grade-Year 1 , 7 th grade-Year 1 , and 7 th grade-Year 2 cohorts, 
respectively. In Mathematics, the respective effect sizes were +0.582, +0.60, and 
+0.38. In two of the MAHS cohorts (6 th grade-Year land 7 th grade-Year 2), significantly 
higher percentages of MAHS students than control students scored Proficient or 
Advanced in one or more subjects. 
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Memphis Academy of Science and Engineering (MASE) 

Matched-Pair Samples 

MASE focuses on preparing high school students for science and engineering 
careers via emphases on science, mathematics, and technology. In its second year of 
operation, it enrolled 124 students in grade 7 and 91 students in grade 8. In the Year 1 
matched-pairs achievement study, 134 matched seventh-grade charter-control student 
pairs were established. Of these, 1 1 dropped out of the school district, 17 transferred to 
another location, 5 were eliminated as a result of their matched control counterparts 
being retained, and 2 were eliminated as a result of their being retained. The remaining 
longitudinal sample for the Year 2 (2004-2005) eighth-grade analysis thus consisted of 
99 matched pairs from Year 1 . In addition, 25 new eighth graders enrolled in MASE in 
2004-2005 and were matched to control-school counterparts. A single pair differed on 
pretest school enrollment. For the seventh-grade analysis, all 91 new enrollees were 
matched to control-school counterparts. Thirty-two pair differed on pretest school 
location. 

Reading and Mathematics Outcomes 

Pretest NCE scores in Reading/Language Arts and Mathematics were compared 
between groups for the above three MASE cohorts. As summarized in Table 2, the 
MASE and control students performed nearly identically on all tests, with all effect sizes 
approximating zero and all ANOVA outcomes nonsignificant. Both eighth-grade cohorts 
(MASE and matched controls), but particularly the second-year enrollees, scored above 
the national norm of 50 in Reading/Language Arts and Mathematics, with the mean 
NCE’s ranging from 51 .92 to 60.31 . The seventh-grade cohorts, however, scored 
slightly below the national norm. Pretest-posttest correlations by grade-level and 
subject were moderately strong to strong, ranging from r= +0.694 to +0.810 (see Table 
3). 

For the analysis of program effects on posttest scores, a multivariate analysis of 
covariance (MANCOVA) was performed, separately by cohort, on the 2004-2005 CRT 
TCAP/AT Reading/Language Arts and Mathematics subtests. Posttest means, 
standard deviations, and effect sizes are summarized in Table 4. 
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Posttest outcomes for 7 th grade-Year 1. The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s <.001 ). 
The multivariate program effect was also significant, F{ 2,177) = 10.92, p <.001 . Follow- 
up univariate ANCOVAs revealed results significantly favoring MASE students in 
Reading/Language Arts, F( 1 ,178) = 21 .95, p <.001 , and directionally favoring them in 
Mathematics, F(1 ,178) = 3.20, p < .08. Adjusted effect sizes were +0.38 in 
Reading/Language Arts, indicating a moderate to strong effect, and +0.14 in 
Mathematics, reflecting a weak effect. 

Posttest outcomes for 8 th grade-Year 1. The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s < .001). 
The multivariate effect of program was also significant, F(2,45) = 5.07, p = .01 . Similar 
to the seventh-grade results, the univariate ANCOVAs revealed significant results 
favoring MASE students in Reading/Language Arts, F(1 ,46) = 8.30, p = .01 , but only a 
nonsignificant directional advantage in Mathematics, F(1 ,46) = 3.63, p = .06. Adjusted 
effect sizes for Reading/Language Arts and Mathematics were +0.52 and +0.27, 
respectively, reflecting moderate to strong effects (see Table 4). 

Posttest outcomes for 8 th grade-Year 2. The Reading/Language Arts and 
Mathematics covariates were both highly significant in the MANCOVA (both p’s < .001). 
The multivariate program effect was also significant, F( 2,193) = 13.20, p < .001 . 
Significant univariate effects favoring MASE students were obtained in both 
Reading/Language Arts, F(1,194) = 7.67, p= .01, and Mathematics, F(1,194) = 26.17, p 
< .001 . Adjusted effect sizes for Reading/Language Arts and Mathematics were +0.24 
and +0.43, respectively, reflecting moderate to strong effects (see Table 4). 

TCAP/AT proficiency levels. Chi-square analyses of the percentages of MASE 
and control students who scored at Below Proficient , Proficient, and Advanced levels on 
the 2004-2005 TCAP/AT subtests were significant for the eighth grade-Year 2 students 
in Mathematics only, and for the seventh grade-Year 1 students in Reading/Language 
Arts only. Higher percentages of the MASE than control students scored Proficient or 
Advanced on these tests. 

Summary of MASE outcomes. For all three cohorts, statistically significant and 
moderate to strong effects (ranging from ES = +0.24 to +0.52), were obtained in 
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Reading/Language Arts, an area strongly emphasized by the school’s curriculum for 
grades 7 and 8. The advantages in Mathematics were significant only for the eighth 
grade-Year 2 cohort (ES = +0.43). For the seventh grade-Year 1 (ES = +0.14) and the 
eighth grade-Year 1 ( ES = +0.27) cohorts, only modest directional advantages were 
indicated. However, the seventh grade-Year 1 MASE students were significantly more 
likely than control students to perform at Proficient and Advanced levels in Reading/ 
Language Arts. The eighth grade-Year 2 MASE students significantly surpassed the 
control group in scoring at Proficient! Advanced levels in Mathematics. 
Smithson-Craighead Academy (SCA) 

Matched-Pair Samples 

The focus of Smithson-Craighead Academy is to meet the social and academic 
achievement needs of at-risk students. In its second year of operation, its enrollment 
consisted of approximately 150 students in five grades — Kindergarten through fourth. 
Pretest scores on TCAP:AT were available for the majority of third and fourth graders 
only, thereby limiting the analyses to those two grades. Due to the relatively small 
number of enrollees per grade level and the sampling attrition associated with creating 
matched pairs of SCA and control students, we conducted the analyses using all 
available matched pairs per grade, combining both “cohort” students (those continuing 
from 2003-04) and new students (initially enrolling in 2004-05). 3 

As with the other charter schools evaluated, SCA-control student matching 
variables included grade level, race, gender, lunch status, special education program 
membership, ELL status, and prior TCAP achievement. Specifically, the latter, 
achievement pretest indices included: (a) the mean NCE scores on the 2003-04 
Reading, Language, and Mathematics subtests of the NRT assessment on TCAP/AT for 
grade 2 students, and (b) the mean number correct on the 2003-04 Reading/Language 
Arts and Math CRT TCAP/AT for grade 3 students.. Prior year school location was not 
included as a matching variable. For grades 2 and 3, it was possible to derive 27 and 
26 matched pairs, respectively. 



3 Separate analyses for these subgroupings are being conducted for exploratory purposes and will be included in an 
addendum to this evaluation report. 
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Reading and Mathematics Outcomes 

Pretest NCE and number correct scores were compared between groups for the 
third- and fourth-graders, respectively. As summarized in Table 2, the SCA and control 
students performed nearly identically on all tests, with minimal effect sizes and all 
ANOVA outcomes nonsignificant. Noting the NCE means for third grade, both the SCA 
and control students, with M’s ranging from approximately 30 to 40, scored well below 
the national norm of 50. These outcomes corroborate the academically at-risk status of 
the student populations concerned. As may be seen in Table 3, the pretest-posttest 
correlations for grade-level by subject ranged from r= +.323 to +0.670. 

For the analysis of program effects on posttest scores, a multivariate analysis of 
covariance (MANCOVA) was performed, separately by grade level, on scale scores 
from the 2004-2005 CRT TCAP/AT Reading/Language Arts and Mathematics subtests. 
Posttest means, standard deviations, and effect sizes are summarized in Table 4. 
Covariates were the pretest scores described above for the respective grade levels. 

Posttest outcomes for 3 rd grade. The MANCOVA performed on 
Reading/Language Arts and Mathematics was not significant , F(2,51) = 0.638, p = 

.533. As shown in Table 4, the SCA and control group means were very comparable, 
while showing a small directional advantage for SCA students in Reading/Language 
Arts (adjusted d = +0.230) and for control students (adjusted d = -0.128) in 
Mathematics. 

Posttest outcomes for 4th grade. The MANCOVA performed on Reading/ 
Language Arts and Mathematics was not significant , F(2,49) = 1 .320, p = .277 . 
However, SCA students demonstrated a small directional advantage in both 
Reading/Language Arts (adjusted d= +0.181) and in Mathematics (adjusted d = 
+0.465). As shown, in Table 5, there were no significant trends in the percentages of 
SCA and control students who scored at Proficient and Advanced levels on the 
TCAP:AT subtests. 

Summary of SCA outcomes. For both grades 3 and 4, no significant differences 
between SCA and matched control students were found. SCA students, however, 
demonstrated directional advantages in Reading/Language Arts in both grades and in 
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Mathematics in grade 4. It should be noted that the in each grade, relatively small 
sample sizes reduce the power for detecting program effects. 



Summary and Conclusions 

Overall, the analyses of achievement indicate mostly positive effects for the 
second-year charter schools but equivocal outcomes for the first-year schools. 
Regarding the latter, only Star Academy but neither of the two high schools 
demonstrated significant advantages or strong effect sizes relative to the control group. 
In the implementation study report (Ross, McDonald, & Bol, 2005), strengths and 
weaknesses of these charter school programs in their first year of operation are 
discussed. However, separate from programmatic issues, it seems likely that the 
assessment of achievement effects may be less sensitive at the secondary level than in 
earlier grades because of the nature of the Gateway tests, which are highly subject 
specific (i.e., explicitly associated with Algebra I, English 10, and Biology) and can be 
taken multiple times by students. Particularly at the high school level, a charter school’s 
success in providing effective teaching and learning may be reflected more meaningfully 
across a wider range of subjects and broader educational outcomes such as 
attendance, graduation rates, and college enrollments. These indices will be examined 
in supplementary analyses. 

Results for the second-year schools were relatively consistent and clearly 
positive. For the three Memphis schools, out of 18 school grade-level cohort x 
TCAP/AT subtest analyses (see Table 4), all effect sizes were positive, with median 
ES ’ s of +0.38 in Reading/Language Arts and +0.43 in Mathematics. Effect sizes of 
these levels would be considered educationally strong and meaningful by researchers. 
Further, of these 1 8 analyses, 1 2 were statistically significant at p < .05. The implication 
is that students attending these charter schools are performing higher than their peers 
who remained in traditional schools. For the Nashville school (Smithson-Craighead 
Academy), no statistically significant effects were found in Reading/Language Arts or 
Mathematics either of the grades examined (3 rd and 4 th ), and the effect sizes were 
minimal. However, SCA students showed slight directional advantages over control 
students in three out of the four grade-level x subject comparisons. 
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As indicated in the first-year report (Ross, McDonald, & McSparrin-Gallagher, 
2005), we encourage readers to interpret the results cautiously given that because of 
student choice and other constraints, we were unable to conduct a randomized 
experimental study that eliminated family interest or involvement as an influential factor; 
and (c) some grade-level matched-pair sample sizes (e.g., in COSLA, Smithson- 
Craighead, and Star Academy) were small and thus subject to sampling error. 

To capsule the achievement outcomes found in this study, a brief achievement 
profile of each of the six schools is provided below: 

First-Year Schools 

City University School of Liberal Arts. Only Algebra I could be analyzed; no 
effects or trends were found. 

Star Academy. Significant or suggestive positive effects were found in both 
Reading/Language Arts and Mathematics. 

Yo! Academy. No effects were found in English 10. No effects were found in 
Algebra I in grade 1 1 , but a nonsignificant disadvantage in Algebra I was indicated in 
grade 1 0. 

Second-Year Schools 

Circles of Success in Learning Academy (COSLA). Significant positive effects 
were evidenced in grade 1 Reading/Language Arts and Mathematics, and suggestive 
positive effects in grade 2 Mathematics. No effects were found in grade 3. 

Memphis Academy of Health Sciences (MAHS). Significant and moderate-to- 
strong positive effects were found across all grades 6 and 7 cohorts in both 
Reading/Language Arts and Mathematics. 

Memphis Academy of Science and Engineering (MASE). Significant or 
suggestive positive effects were found across all grade 7 and 8 cohorts in both 
Reading/Language Arts and Mathematics. Effects were relatively more pronounced in 
Reading/Language Arts than in Mathematics, and for the eighth grade-Year 2 cohort 
than for the Year 1 seventh and eighth grade cohorts. 

Smithson-Craighead Academy. No reliable effects were found in either 
Reading/Language Arts or Mathematics in grade 3 or grade 4. Slight directional 
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advantages were evidenced for SCA students in Reading/Language Arts in both grades 
and in Mathematics in grade 4. 
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